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Abstract 

IMPACT is a media and technology program designed to support and promote effective 
instruction through the integration of technology. The model is currently being implemented in 
30 K-12 schools throughout North Carolina. This evaluation study investigated the effects of 
this model on the attitudes, behaviors, and activities of students and teachers. Information about 
the measures, data collection/analysis methods and implementation issues will be addressed. 

Objective 

The objective of this study was to evaluate the longitudinal effects of a large-scale 
technology integration model on student and teacher outcomes, including student achievement 
and technology skills, teacher use of technology during instruction, and instructional activities. 

Perspectives 

Numerous studies have suggested that access to technology in the classroom is associated 
with higher student achievement (Christmann, 1997; Mann & Shafer, 1997; Page, 2002; 
Wenglinsky, 1998). However, these effects differ across studies and across populations and 
subject areas (e.g., Heineke, Blasi, Milman, Washington, 1999). Few studies have investigated 
the effects of a comprehensive technology integration model, but available data indicate that 
such models hold more promise for improving academic achievement (e.g., Mann, Shakeshaft, 
Becker, & Kottkamp, 1998). According to ISTE, a number of conditions are essential to create 
learning environments in which technology may be integrated successfully; these include 
administrative support and vision, student-centered approaches to learning, access to 
contemporary technologies, technical assistance for maintaining and using technologies, and 
ongoing financial support for sustained technology use (ISTE, 2007). Similarly, teachers' 
effective use of technology in the classroom is associated with a number of factors: appropriate 
staff development, opportunities for collaboration with colleagues, modeling of best practice 
(Burns, 2002), and institutional support (Darling-Hammond & McLaughlin, 1995; Guskey & 
Sparks, 1996). Reflecting these standards, the IMPACT model (NC DPI, 2008) provides an 
innovative framework for infusing technology and media resources into instructional programs, 
focusing on the collaboration among teachers and technology and media personnel, as well as 
appropriate and sustained staff development and support. 

The IMPACT program follows the guidelines for outstanding media and technology 
programs included in the state’s Educational Technology Plan. This model is also aligned to 
Information Power: Building Powerful Partnerships, the national standards for media and 
technology programs. In addition, the staff development provided to school personnel through 
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this model is based on the NSDC Standards for Professional Development, which are based on 
research that demonstrates the link between staff development and student learning (Hirsh, 

2006). A primary goal of this project is to produce technologically literate students, consistent 
with the TAAP report, which states that students should have a broad perspective on the uses, 
variations, and purposes for technology use (TAAP, 1996). 

The IMPACT model for technology integration has, at its core, collaboration between 
teachers in conjunction with the technology facilitator and media coordinator. This collaboration 
emphasizes the best uses of technology and media collections matched to content standards with 
the goal of improved student achievement. The model promotes school leadership that includes 
the principal, technology facilitator, and media coordinator, fostering a collaborative 
environment for all stakeholders. The IMPACT Model also requires schools to offer flexible 
access to media/technology resources, also referred to as “flexible scheduling.” Professional 
development was a required component, though the format was left up to individual schools. In 
addition, all IMPACT schools were expected to hire a full-time technology facilitator (TF) to 
lead the integration of instructional technology in the school. For more information about the 
model’s components, see http://www.ncwiseowl.org/impact/ 



Methods 

The primary goal of the IMPACT evaluation was to measure whether the implementation 
of the IMPACT model makes a significant difference in student achievement. Given the 
encompassing nature of the IMPACT framework, we anticipated that this model would also 
affect other student and teacher outcomes. The following discussion describes the population, 
study design, and measures used to assess a selection of the variables studied in the IMPACT 
evaluation from 2003-2009. 



Population 

All IMPACT schools are Title I (low SES) sites; 25% are located in urban districts, with 
the others located in rural areas. The IMPACT I cohort (2003-06) included 1 1 schools, including 
grades K-8. The IMPACT II cohort (2007-09) included 5 middle schools. In July 2008, funds 
were awarded to 7 additional school districts, including 30 schools (elementary, middle and 
high) that have a high percentage of economically disadvantaged students. Three of the school 
districts participating in the 2006-07 IMPACTing Leadership training received funding in July 
2008 as the IMPACT III cohort. Four additional school districts were selected through a 
competitive RFP process that took place in Fall 2007; they received funding in July 2008 as the 
IMPACT IV cohort. A unique feature of the third and fourth cohorts is the district’s role in 
planning and implementation. In these latter cohorts, applications were accepted from districts, 
as opposed to individual schools, with the district applicants planning to implement the IMPACT 
model at most/all schools within their district. 
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Design and Data Sources 



A quasi-experimental (matched subjects) longitudinal design was used to assess the 
effects of the model for the IMPACT I and IMPACT II cohorts. Analysis of Variance, Logistic 
Regression, as well as hierarchical linear modeling (HLM) techniques were used to assess 
changes in teacher and student variables over time. The evaluation design was multivariate, 
assessing an extensive number of variables across multiple levels (teacher, student, 
administrator, school). The following discussion describes selected measures used to assess 
some of these variables. 

Changes in teachers’ technology use and skills were evaluated with a survey based on the 
NETS-T standards (Altee, 2001; ISTE, 2002). This survey assesses six constructs, including 
Technology Operations and Concepts (10 items). Planning and Designing Learning 
Environments and Experiences (8 items). Teaching, Learning, and the Curriculum (6 items). 
Assessment and Evaluation (6 items). Productivity and Professional Practice (7 items), and 
Social, Ethical, Legal, and Human Issues (1 1 items) (Corbell, Osborne, & Grable, 2008). Each 
item begins with the prompt “In your experience as a teacher do you. . and is answered via a 
four-point Likert-scale ranging from 0-Not at alt, Minimally (need help); 2- confidently 
(knowledgeable and fluent); and 3- Able to teach others. Confirmatory factor analyses (Corbell, 
Osborne, & Grable, 2008) have determined that the 48 survey items could be analyzed according 
to the six-factor model (all a > 0.93), or in a simpler single-factor model. 



Teachers’ instructional activities were measured using a self-report questionnaire. 

Teacher Activities of Instruction (AOI) (Grable & Park, 2002). Adapted from a survey used in 
the NSE Systemic Change Initiative (Horizon Research, 1996), this questionnaire was used to 
measure classroom activities used during a specified four-week period of the spring semester, 
based on the argument that when teachers have the opportunity to reference their lesson plans, 
self-report data about their instructional activities has a higher degree of accuracy (Davidson, 
2000; Grable & Park, 2002). Items addressing a wide range of possible classroom activities 
were included in order to gauge constructivist practices in the classroom, one tenet of the 
IMPACT model. Responses for frequency of use range from 0- ‘never” to 5-“almost every 
day.” Confirmatory factor analyses determined that a 20-item subgroup of these items was the 
best factor solution for the survey; this single factor was stable across teacher populations from 
different school levels (all a>.93) (Grable, Overbay, Seaton, Shattuck, & Osborne, 2008). 

Within the context of the IMPACT evaluation, items addressing computer use were added as a 
separate section on the survey (a=.80). 

The quality of the professional development was assessed using the Standards 
Assessment Inventory developed by the National Staff Development Council (SEDL,2003). 

This inventory was administered to all sehool staff in April 2009 and 2010, and addresses twelve 
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standards, combined into three key categories (Context, Process and Content) found to be 
essential to effective professional development (Hirsch, 2006). 

Students’ technological literacy was measured with a Technological Skills checklist 
administered to all IMPACT students. The Technology Skills Checklist (TSC) was originally 
designed by Berkeley Planning Associates in conjunction with the Teacher Led Technology 
Challenge project (Berkeley Planning Associates, 1998). This survey measures a number of 
variables, but prominently features a set of items (32 items on the grades 6-8 version, and 27 
Items on the 3-5 version) that address students’ perception of their technology skills. The five- 
point scale includes the following responses: 1 do not know if I have done this,” 2-“/ have 
never done this,” 3- ‘I can do this with some help,” A- ‘I can do this by myself,” and 5- ‘I can show 
someone how to do this.” Confirmatory factor analyses indicated that for the TSC 6-8, a four- 
factor model provided the best fit for the data: File maintenance (a = 0.89), Database operations 
(a = 0.89), Basic technology skills (a = 0.88), and Advanced technology skills (a = 0.91). On 
the TSC 3-5, a three-factor model provided the best fit for the data: File maintenance (a =.82), 
Basic computing skills (a=.83), and Advanced technology skills (a=.80). 

Student achievement was assessed using standardized state-administered assessments for 
North Carolina, which include End-of-Grade tests in grades 3-8, and End-of-Course tests in 
grades 9-12, both of which are criterion-referenced tests. More information about the 
development, scoring, and administration of these tests can be found at 
http://www.ncpublicschools.org/accountability/ 



Results 

Where possible, the discussion of survey results focuses on results from IMPACT I and 
II, cohorts in which Comparison students and teachers completed surveys, as the ability to 
examine within and between-subject differences over time provides more information about the 
comparative effects associated with the model. Student achievement is discussed within the 
context of the IMPACT IIFIV evaluations, as these cohorts were larger and more comprehensive 
(including high schools), and provide more information about the effects associated with the 
model in its most current iteration, the district-wide approach. 



Teachers’ Technology Skills (NETS-T) 

Results from all four cohorts of the IMPACT evaluation indicated that teachers made 
significant growth over time in their perception of their technology skills. Over the initial 2-3 
years of implementation, survey results from all cohorts showed significant growth in teachers’ 
confidence in their ability to implement the National Educational Technology Standards, as 
measured by the NETS-T survey. In the IMPACT I study (2003-2006), HEM analyses 
indicated that IMPACT I teachers started the project significantly less confident about their skills 
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than Comparison teachers, but showed significant and substantial growth in their self-perceived 
technology proficiency, whereas there was no significant growth in Comparison school teachers’ 
self-perceived technology proficiency (p for IMPACT < .0001, for IMPACT * time < .0001). In 
IMPACT II (2007-2009), repeat-measures Analysis of Variance results indicated a small but 
statistically significant effect for the interaction between time and group, indicating that the 
IMPACT II teachers made significantly greater improvement in their self-reported technology 
skills, relative to Comparison teachers (mF( 2 , 2 i 3 ) =4.09, p<.05, eta^=.04). Figure 1 provides a 
graphical illustration of these data. As this figure shows, IMPACT II teachers began the project 
with significantly lower NETS-T scores, but by Year 2 of the implementation process had pulled 
even with Comparison teachers; their current trajectory indicates that IMPACT II teachers’ 
perceived technology skills are following a trend similar to the one experienced by IMPACT I 
teachers in the initial iteration of the model. 

Figure 1. IMPACT II teachers’ NETS-T skill scores, Year 1-Year 2 




Activities of Instruction (AOI) 

One aspect of the IMPACT evaluation involves examining the types of instructional 
approaches used within a technology-rich environment. In particular, evaluators have been 
interested in changes in the use of constructivist approaches reported by classroom teachers. 
Results for the AOI Constructivism scale indicated that there was a small but significant effect 
for time*group (mF( 2 ,i 87 ) =3.29, p<.05, eta^=.03); this effect was largely due to the decline in the 
level of constructivism reported by Comparison teachers between Year 1 (post) and Year 2 
(post); overall, the level of constructivism reported by IMPACT II teachers remained essentially 
stable. 
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There was a much larger effect for technology use; IMPACT II teachers made 
significantly greater strides in their reported use of technology, relative to Comparison teachers, 
whose reported use of technology declined during the project (mF( 2 , 195 ) =14.64, p<.0001, 
eta =.13). Figure 2 provides a graphical illustration of these results. Interestingly, IMPACT II 
teachers reported a slight decline in the frequency of technology use in the classroom in Year 2; 
because access to technology and exposure to professional development (key elements of the 
model) continued to increase in Year 2, this finding may reflect IMPACT II teachers’ 
willingness to make judgments about (in)appropriate uses of technology to enhance instruction. 

Figure 2. IMPACT II teachers’ Activities of Instruction (AOI) scores, Year I -Year 2 
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NSDC’s Standards Assessment Inventory (SAI) 

The SAI was administered to all IMPACT III /IV schools in Spring 2009 \ in an effort to 
measure the prevalence/intensity of practices found to be indicative of high quality professional 
development programs, and reflective of the twelve standards for staff development outlined by 
the NSDC, which are used as a basis for the North Carolina Standards for Professional 
Development. Results from analyses of IMPACT IIFIV schools were consistent 
with results obtained from the larger national dataset used in the factor analysis (N=5 1,000), in 
that, average ratings for each factor differed significantly from each other within both of the 
IMPACT cohorts. In fact, significant differences were noted between all pairs of factors at the 
.001 level, with the exception of the School Leadership/Equity pair within the IMPACT IV 
cohort, in which no significant difference was observed (p=.235). 



* The SAI was not administered to the IMPACT I or II cohorts. 
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Table 1. Standards Assessment Inventory (SAI). Average Ratings by Factor (Scale of 0-4) 



IMPACT Model /Factor 


N 


Mean 


SD 


IMPACT III 








Opportunities for PD/ 
Teacher Collaboration 


484 


2.84 


.72 


School Leadership 


484 


3.09 


.68 


Equity 

Teacher Influence on 


484 


3.20 


.58 


Decisions/Policy 


484 


2.59 


.77 


IMPACT IV 








Opportunities for PD/ 
Teacher Collaboration 


553 


2.91 


.68 


School Leadership 


553 


3.15 


.69 


Equity 

Teacher Influence on 


553 


3.17 


.57 


Decisions/Policy 


553 


2.62 


.80 



It is important to note that average ratings for the School Leadership and Equity factors 
were significantly higher than those for the other two factors. Again, this is consistent with the 
previously cited technical report, which included a sample of over 51,000 teachers in four other 
states, thus indicating our results are reflective of a more widespread phenomenon, extending 
beyond the schools in this project. These authors suggest that “the kind of job-embedded 
collaborative learning that has been found to be important in promoting instructional 
improvement and student achievement is not a common feature of professional development 
across many schools. In addition, teachers’ lack of influence over school decisions means that 
teachers are less likely to be engaged in collaborative problem-solving around school-specific 
issues” (p.55). 

However, other findings that emerged from the SAI indicated that 37% of teachers in the 
IMPACT III and IV cohorts reported they frequently /always observe each other’s classroom 
instruction as one way to improve teaching, and 54% reported they frequently/always received 
feedback from colleagues about classroom practices. Of those surveyed, 79% said teacher 
learning was frequently/always supported through a combination of strategies (e.g. workshops, 
peer coaching, study groups, joint planning of lessons, and examination of student work). 
Further, 67% reported they frequently/always receive support implementing new skills until they 
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become a natural part of instruction, and 49% said they frequently/always get to choose the kind 
of professional development they receive. 



Student Technology Skills: Tech Skills Checklist (TSC) 

Results from the IMPACT I study indicated that IMPACT I students in grades 3-5 
experienced stronger growth in their perceived technology skills, but there was not a significant 
difference in the amount of growth on this variable for students in grades 6-8; IMPACT I middle 
school students had slightly higher average scores on this variable, but did not experience a 
significantly different growth pattern, relative to Comparison students. In the IMPACT II 
project, a somewhat different pattern was observed. Results from the TSC 6-8 survey 
administered to IMPACT II middle school students and their Comparison counterparts indicated 
that Comparison students had higher average perceptions of their technology skills at all 3 
timepoints measured. However, the interaction between time*group was significant; IMPACT II 
students experienced more growth on this variable over time (mF( 8 , 2408 )=l 1-40, p<.0001, 
eta^=.01). This effect was strongest for the “Database” factor (F(2,320)=35.28,p<.0001, eta^=.02); 
these results are illustrated in the figure below. 

Figured. IMPACT II students’ TSC 6-8: “Database” factor scores. Year 1-Year 2 




Student Achievement: EOG/EOC Test Scores 
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The findings from IMPACT I demonstrated a significant association between 
implementation of the IMPACT Model and student achievement; IMPACT I students 
experienced stronger growth on Reading and Math scale scores, and were more likely to move 
from failing to passing on their state-administered End-of-Grade tests (Overbay, Osborne, 
Seaton, Vasu, & Grable, 2007). IMPACT II schools, have not shown the same pattern, to date, 
although analyses of the current school year’s test data are pending (Overbay & Vasu, 2010). 
The IMPACT III and IV cohorts are larger and more comprehensive in scope than either the 
IMPACT I or II cohorts; the following discussion describes selected findings associated with 
these two iterations of the IMPACT model. 

IMPACT III: Student achievement results 

There were 7,244 students included in these analyses who took an End-of-Grade test 
while enrolled at an IMPACT III or matched Comparison school in 2009. The groups included 
3,793 enrolled at IMPACT III elementary or middle schools, and 3,451 students enrolled in 
Comparison schools. The Comparison schools were selected because their demographics and 
achievement were similar, at baseline, to the IMPACT III schools. The table below illustrates 
the primary variables on which they were matched. 

Table 2. Demographic characteristics of IMPACT III and Comparison schools (gr.3-8) 





IMPACT III 


Comparison 


Total number of students enrolled in 2009 
Demographics- 


3,793 


3,451 


White 


45.4% 


46.9% 


Black 


39.8% 


34.1% 


Other 


14.8% 


19% 


Economically Disadvantaged 


57.5% 


59.1% 


Students with Disabilities 


14.4% 


10.9% 



IMPACT III: Changes in Reading achievement in Years 1-3. Because 3 time points were 
available for this group, HEM analyses were used to examine changes in the DV (Reading scale 
score) over time. The null/fully unconditional model was conducted to partition the variance 
between each level of the model and to ensure that there was sufficient variability at Level 1 and 
Level 2 to warrant continuation with analyses (Raudenbush & Bryk, 2002). Only the dependent 
variable, Reading scale score, was entered in this model. Results from this analysis indicated 
that 76% of the variability in Reading scale score was between-students (too = 1 12.63, z = 49.21, 
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p < .0001) and 24% was within students (a^ = 36.44, z = 66.11, p < .0001). Therefore, the 
null/fully unconditional model indicated that there was sufficient variability to warrant further 
analyses. 



A subsequent model was used (Means-as-Outcomes regression) to measure the 
relationship between the school in which a student was enrolled (IMPACT school v. 

Comparison school) and their Reading scale score. Specifically, this model measures whether 
there were any between-student differences in Reading scale scores. Also, it measures how 
much between-student variability is accounted for by knowing whether the student attended an 
IMPACT III school or a Comparison school. There was a significant relationship between 
whether students were enrolled at an IMPACT III or Comparison school and their average 
Reading scale score. On average, students in IMPACT III schools scored higher than students in 

Comparison schools. (Yoi = 2.382, t=8.64,p<.0001) However, this model accounted for only 
1 % of the between-student variability in Reading scale score. 

The next model attempted to determine if the change in Reading scale score over three years 
was dependent on students’ project school enrollment. Results showed no significant project 
school enrollment x administration year interaction, such that the change/growth over time in 

Reading scale score was unrelated to the students’ project school enrollment (Yn= -0.0585, t= - 
0.61 , p=.5424). Additional models indicated that there was no significant interaction between 
year x project school enrollment x free/reduced lunch status, or between year x project school 
enrollment x SWD, indicating that the relationship between growth over time and project school 
enrollment was not significantly related to either students’ ED status or disability status. 

When school level was added to the equation, results indicated that growth in Reading scores 
over time simultaneously depended on project school enrollment and school level (i.e. school 

level X year x project school enrollment Y 13 = .8539, t= 4.18,p<.0001). The largest average 
change in Reading scores was among students enrolled in IMPACT middle schools. 

Specifically, middle school students in IMPACT III schools showed more growth in Reading 
scores (avg. increase of 7.45 pts) than middle school students in Comparison schools (avg. 
increase of 5.61 pts). This model accounted for 50% of the within-student variability and 8 % of 
the between-student variability. This finding may indicate that the effects of instructional 
technology on Reading achievement are more pronounced with older students, but further 
investigation is required. 

IMPACT III: Changes in Math achievement in Years 1-3. The null/fully unconditional 
model was conducted to partition the variance between each level of the model. Only the 
dependent variable. Math scale score, was entered in this model. Results from this analysis 
indicated that 72% of the variability in Math scale score was between-students (tqo = 84.932, z = 
48.27, p < .0001) and 28% was within students (a^ = 32.744, z = 67.31,p < .0001). Therefore, 
the null/fully unconditional model indicated there was sufficient variability to warrant further 
analyses. 
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A subsequent model was used (Means-as-Outcomes regression) to measure whether there 
were any between-student differences in Math scale scores, based on their enrollment in an 
IMPACT III school or a Comparison school; this model indicated that there was a significant 
relationship between their school enrollment (IMPACT v. Comparison) and their average Math 
scale score. On average, students in IMPACT III schools scored higher than students in 

Comparison schools (Yoi = 2.364, t=9.78,p<.0001). However, this model accounted for only 
2% of the between-student variability in Math scale score. 

The next model assessed whether the change/growth in Math scale score over three years was 
dependent on students’ project school enrollment. Results showed a significant school 
enrollment x year interaction, such that the change/growth over time in Math scale score was 

dependent on the students’ project school enrollment (Y n= 0.4223, t= 4.72, p<.000l). 
Specifically, students enrolled in IMPACT III schools showed slightly stronger growth (avg. 
increase of 5.2 points) in Math scores than students enrolled in Comparison schools (avg. 
increase of 3.85 pts). 

Using Logistic Regression techniques to examine Math achievement levels (I - IV) 
revealed that IMPACT III students tended to show more improvement than their Comparison 
counterparts. This is consistent with previously reported findings from multilevel analyses. 
Looking at the change in achievement level from 2007 (baseline) to 2009, we see that IMPACT 
III students were 42% more likely to increase achievement levels (OR=1.593,p=.000). This 
finding, in particular, is consistent with findings from the IMPACT I cohort, including only 
elementary schools from 2003 to 2006, in which it was observed that IMPACT I students were 
37% more likely to increase performance levels in Math from baseline to Year 2 (Osborne, et ah, 
2007). 

To further illustrate this difference, binary logistic regression was used to measure differences 
in the odds of passing or scoring above grade level between IMPACT III and Comparison 
students, in 2007 (baseline) and 2009 (Year 2). The outcome variables were dichotomous - pass 
(Level 3-4)/fail (Level 1-2) and Level 4/ Level 1-3. While IMPACT III and Comparison 
students were equally likely to pass the Math LOG in 2007, IMPACT III students were 
significantly more likely to pass, and score above grade level on the Math LOG in 2009 

(p<.001). 

Student Achievement Outcomes - IMPACT IV 

There were 9,427 students included in these analyses who took an End-of-Grade test while 
enrolled at an IMPACT IV or matched Comparison school in 2009. The groups included 4,977 
enrolled at IMPACT IV elementary or middle schools, and 4,450 students enrolled in 
Comparison schools. The Comparison schools were selected because their demographics and 
achievement were similar, at baseline, to the IMPACT IV schools. The table below illustrates 
the variables on which they were matched. 
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Table 3. Demographic characteristics of IMPACT IV and Comparison schools (gr.3-8) 





IMPACT IV 


Comparison 


Total number of students enrolled in 2009 
Demographics- 


4.977 


4,450 


White 


44.6% 


50.4% 


Black 


31.2% 


24.4% 


Hispanic/Latino 


18.6% 


18% 


Economically Disadvantaged 


65.8% 


65.4% 


Students with Disabilities 


12% 


11.3% 



Changes in Math and Reading achievement - Year 1 to Year 2. Because only two 
timepoints were available for analysis (the IMPACT IV cohort began implementation in 2008), 
logistic regression was used to analyze differences in IMPACT IV and Comparison students’ 
odds of passing between Year I and Year 2. Results indicated that the odds of passing the Math 
EOG test did not change significantly from 2008 to 2009 for either IMPACT IV schools or their 
matched Comparison schools. However, the year of administration was a significant predictor of 
whether students would pass the Reading EOG at IMPACT IV schools. The odds of an 
IMPACT IV student passing the Reading EOG test in Year 2 were 1.133 times their odds of 
passing in Year 1 (p = .004). This same pattern was not observed in Comparison schools. 
Specifically, IMPACT IV students were 7% more likely to pass Reading in 2009 than they were 
in 2008. The odds ratios, relative risks, and p-values for both groups can be found in Table 4. 



Table 4. Odds of passing EOG tests in 2008 and 2009 for IMPACT IV & Comparison schools 





Odds Ratio 


Relative Risk 


p-value 




Comparison 


1.054 


1.02 


.261 


Math 


IMPACT IV 


1.074 


1.03 


.105 




Comparison 


1.058 


1.03 


.204 
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Reading 



IMPACT IV 



1.133 



1.07 



.004 



Additional models suggest that, to date, the IMPACT IV model seems to have had a more 
noticeable impact on Economically Disadvantaged (ERL) students. Analysis of the passing rates 
of ERL students on the Math and Reading LOG test determined that the year of administration 
was a significant predictor for IMPACT IV schools, but not Comparison schools. Specifically, 
the odds of an ERL student at an IMPACT IV school passing the Math LOG increased 
significantly from Year 1 to Year 2 (p = .007). The odds increased slightly for Comparison 
students, though the increase was not significant. The year of administration was also a 
significant predictor of passing rates on the Reading EGG for IMPACT IV schools (OR = 1.192, 
p = .002). Specifically, ERL students in IMPACT IV schools were 9% more likely to pass the 
Reading LOG in 2009 than they were in 2008. The odds ratios, relative risks, and p-values for 
both groups can be found in Table 5. 



Table 5. Odds ofFRL students passing the EOG test at IMPACT IV and Comparison schools 





Odds Ratio 


Relative Risk 


p -value 




Comparison 


1.117 


1.07 


.053 


Math 


IMPACT IV 


1.156 


1.10 


.007 




Comparison 


1.095 


1.04 


.111 


Reading 


IMPACT IV 


1.192 


1.09 


.002 



Significance 

The components of the IMPACT model are designed to provide the additional support 
and scaffolding necessary to leverage the technology in valuable ways, ultimately improving 
student learning experiences. The model includes strong leadership and support from the 
administration as essential components for successful implementation, a common feature of 
many educational innovations and initiatives being introduced in a K12 setting. Eurther, the 
emphases on collaborative planning and professional development reflect research-based 
practices intended to create a supportive and enriching environment for the integration of 
technology into instruction. 

Results collected from the four cohorts of the IMPACT evaluation indicate that there are 
a number of interesting effects associated with this model, though these effect vary to some 
extent depending on the cohort measured. The district-wide model used in IMPACT III and IV 
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appears to have a promising association with student achievement, but each iteration of the 
model has seen positive effects on teacher variables such as technology skills and use of 
technology for instruction. Future plans in IMPACT schools involve “taking it to the next 
level,” focusing on improvement in classroom instruction, using more complex technological 
resources, and allowing students to become producers of technological products, rather than just 
consumers of technology. 

It is important to evaluate the contribution of each facet of this model to student learning 
and achievement, as well as changes observed in teachers’ instructional practices. A primary 
goal of the IMPACT model is to produce technologically literate students equipped with the 
knowledge and 21st Century skills necessary to become productive members of their 
communities, so these results would be interesting to practitioners, as well. Further, improving 
technology integration in educational settings is of critical importance, and knowledge of how to 
best evaluate these types of programs will be very useful to many educational researchers. 

This evaluation will continue through the 2010/1 1 school year under the umbrella of the 
IMPACT Continuation project, and findings will be expanded throughout the coming year to 
include changes in college attendance/performance for graduates of IMPACT high schools. 
Specifically, all high schools in the IMPACT III and IV cohorts have received additional funding 
to purchase teacher and student laptops in Spring 2010. The K-8 schools in these cohorts have 
received funding to purchase teacher laptops for everyone, and upgrade existing equipment. 
Given the broad support for the IMPACT model within the state of North Carolina, we anticipate 
that findings from the ongoing evaluation will continue to inform school reform efforts in our 
state. 



Interested session participants may contact the authors or the NC Dept of Public 
Instruction - Instructional Technology Section Chief ( nkimrey@dpi.state.nc.us) for additional 
information. 
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